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Introduction 

The following is an evaluation of a Goals 2000 project being carried out in a rural western school district. 

Project participants have embarked on a series of training modules designed to develop their skill with a 
variety of software applications and the integration of technology within the curriculum. Participants are 
receiving training and a new laptop computer with Windows® 95 for use in their respective jobs. Participants 
include administrators and classroom teachers. 



An initial study was conducted in the fall to frame a benchmark for evaluating the success of the project in 
providing the teachers with the technical skills, knowledge, and motivation necessary to integrate technology 
within the curriculum effectively and efficiently. A follow-up evaluation was conducted the following spring. 
This is an ongoing project, so evaluations will continue to be conducted on a yearly basis. 



Methodology 

Six questions formed the focus of this evaluation; 

1 . What technology related skills do participants in the project already possess? 

2. What concerns do participants currently have regarding the integration of technology within the 
curriculum? 

3. How are teachers currently using technology within the classroom? 

4. At what Level of Use are participants in their current use of technology and its integration within 
the curriculum? 

5. What technology related skills are students currently acquiring through their teachers’ use of 
technology within the curriculum? 
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6. What is the perceived impact of current technology use by teachers on students? 

Data collected to answer these questions were obtained through the use of interviews with, and 
questionnaires completed by, participants in the project. Initially, 23 project participants were interviewed 
and completed the questionnaires: 2 school board members, 4 elementary teachers, 10 secondary teachers, 
and 7 administrators. In the spring follow-up, 14 participants were interviewed and completed 
questionnaires: 6 elementary teachers, 7 secondary teachers, and 1 administrator. The group that 
participated in the initial evaluation was not included in the spring evaluation. We were not prepared at that 
time to do a comparison group based on training, across the group. The district was well represented in this 
project with an administrator and at least two teachers from each building being included. Participants 
included educators who were already well versed in technology and others who had limited experience with 
technology. Participants ranged from 23 to 57 years in age, with the average age of the participants being 43; 
57% of the participants were female and 43% male. Teachers ranged from first-year teachers to those with 31 
years of experience. 

The Concerns-Based Adoption Model (CBAM), a model developed in the 1970s to assess, facilitate, and 
evaluate the change process, served as the basis for this study. CBAM provides a strong tool for evaluating 
implementation efficiently and effectively. The diagnostic components of CBAM are designed to evaluate the 
change and adoption process, attending not only to the technical problems associated with the adoption of an 
innovation, but also to the personal perceptions of the potential adopter. 

The CBAM model includes three diagnostic dimensions: (1) Stages of Concern (SoC), (2) Levels of Use (LoU), 
and (3) an Innovation Configuration Map (ICM) . The Stages of Concern questionnaire (SoCQ) and Levels of 
Use interview were used in this evaluation. The Stages of Concern questionnaire was used to assess the 
intensity of the feelings and perceptions that the participants held in relation to the integration of technology 
within the curriculum. (Alpha coefficients on the SoC questionnaire range from .64 to .83, and test-retest 
correlations range from .65 to .86.)' The Levels of Use interview was used to assess where participants were 
in their current use of technology and its integration within the curriculum. (The LoU interview has a 98% 
correlation with observation.)^ 

After completing the Stages of Concern questionnaire, participants were asked to respond to the open-ended 
question “What are your concerns about integrating technology into the classroom?” The question was used 
to clarify the concerns expressed by project participants. 

In addition to these diagnostic procedures, participants were asked to complete two additional 
questionnaires: an impact survey and a self-assessment of their skills with computers and application 
software. The impact survey contained questions regarding the effect of current technology integration 
within the curriculum. The statements participants were asked to respond to addressed the quality of student 
work, learning behaviors, teaching with technology, communication with parents and students, interactive 
classroom organization, student technology skills, and student interaction. The purpose of the survey was to 
assess the participants’ perceptions of the effect that their current use of technology is having on student 
learning, the development of technology skills by students, student achievement, and teacher pedagogy. 

The self-assessment contained specific questions regarding a variety of technology skills: computer 
operations, word processing, operating systems, spreadsheets, databases, computer graphics, desktop 



' Hall, G. E., George, A., & Rutherford, W. (1 986) . Measuring stages of concern about the innovation: A manual for the use of the SoC 
questionnaire. Austin: The University of Texas at Austin, Research and Development Center for Teacher Education. 

Hall, G. E., & Hord, S. M. (1 987) . Change in schools: Facilitating the process. New York: State University of New York Press. 
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publishing, telecommunications, and multimedia. The self-assessment was used to assess the participants’ 
current skill with these technology applications. 



Data Analysis 

The data collected from each of these assessments were analyzed using a combination of quantitative and 
qualitative methods. A comparison of means was used to analyze the scores obtained from the SoC 
questionnaire. The LoU interviews were scored using a rubric created by the developers of the Concerns- 
Based Adoption Model.^ A frequency count was used to analyze the responses to the impact questionnaire. 
Means calculated within each skill area were used to analyze the self-assessment. The data obtained from 
individual participants were compiled and comparisons made between their responses. Trends were 
identified and outliers explored in greater depth. 

The information was also grouped according to the participants’ roles in the district; administrator, 
elementary teacher (K-6), and secondary teacher (8-12). The data obtained from school board members 
were included with those obtained from the administrators, unless otherwise noted. The data obtained from 
the K-12 music teachers were included with those obtained from the secondary teachers. 



Results 

'What technology related skills do participants in the project already possess? 

The participants in this project, as a whole, have very limited technology skills. In the self-assessment 
questionnaire participants were asked to rate their ability — (1) no-knowledge, (2) limited knowledge, (3) 
some knowledge, (4) competent, and (5) expert — with the following technology skills: computer operations, 
word processing, operating systems, spreadsheets, databases, computer graphics, desktop publishing, 
telecommunications, and multimedia. Most participants rated themselves ranging from some knowledge to 
competent in response to the question “I know how to properly boot up, operate, and shut down the 
computer.” But, otherwise, skills were limited in most areas (see Figure 1). 

Administrators rated their word processing, operating system, telecommunications, and spreadsheet 
knowledge between limited knowledge and some knowledge. Their strongest skill area (see Table 1) , word 
processing (2.83) , was rated as having some knowledge. The administrators rated their skill level closer to no- 
knowledge in the other skill areas: computer operations, databases, computer graphics, desktop publishing, 
and multimedia. 



^Hall, G. E., & Hord, S. M. (1987) . Change in schools: Facilitating the process. New York: State University of New York Press. 
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Figure 1. Technology skills self-assessment. Results are displayed by educator role. 



Secondary teachers rated their level of ability in all skill areas between limited knowledge and competent. 
They rated their strongest skill areas (see Figure 1) between some knowledge and competent: word 
processing (3.34), operating systems (2.81), and spreadsheets (2.82). The secondary teachers rated their skill 
level closer to limited knowledge in the other skill areas: telecommunications, computer graphics, databases, 
desktop publishing, computer operations, and multimedia. 



The elementary teachers rated their level of ability in spreadsheets, computer graphics, and operating 
systems between limited knowledge and some knowledge. They rated their word-processing skills as 
competent (2.89; see Figure 1). The elementary teachers rated their skill level closer to no-knowledge in the 
other skill areas: telecommunications, multimedia, desktop publishing, databases, and computer operations. 



What concerns do participants currently have regarding the integration of technology 
within the curriculum? 

Initially, administrators had high Informational concerns and high Collaboration concerns. The one 
administrator interviewed during the spring meeting demonstrated lower Informational concerns. 
Informational concerns focus on a general awareness about integrating technology within the curriculum and 
an interest in learning more detail about the integration of technology. Collaboration concerns focus on the 
coordination of activities related to the integration of technology with others. These concerns would seem to 
be appropriate for administrators. Board members had very much the same concerns. 
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Table 1. Technology Skills Self-Assessment Mean Ratings 



p ■ 
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Administrator 


Secondary 


Elementary^ 


Computer Operations 


1.37 


1.95 


1.04 


Word Processing 


2.83 


3.34 


2.89 


Operating Systems 


2.14 


2.81 


2.00 


Spreadsheets 


1.85 


2.82 


2.28 


Databases 


1.46 


2.25 


1.35 


Graphics 


1.52 


2.33 


2.01 


Desktop Publishing 


1.45 


2.05 


1.37 


Telecommunications 


1.96 


2.39 


1.57 


Multimedia 


1.18 


1.93 


1.38 



In the initial group, the concerns of the elementary and secondary teachers were more extensive. Elementary 
teachers had high Informational and Collaboration concerns, but they also had high Personal concerns. High 
Personal concerns indicate that the individual is uncertain about the demands of integrating technology 
within the curriculum and the individual’s ability to carry out the task. This could include concerns about the 
reward structure in place for this project, decision-making matters and how they are being carried out, and 
potential conflicts with other areas of the job. Informational concerns were still high in the spring, indicating 
that the problems identified, because of lack of information, in the fall had not been addressed. Management 
concerns were also beginning to peak among the participants, indicating the participants were beginning to 
be concerned about how they would manage the use of technology within their classrooms. Participants 
continued to express Collaboration concerns with an additional emphasis on Refocusing concerns. The 
participants seemed to have ideas for refocusing their efforts on options other than technology that would 
enhance learning in the classroom. The elevated Collaboration concerns would indicate that the participants 
continued in their concern with coordinating their efforts with those of other teachers in their buildings. This 
could be emphasized because the schools included within the study offer specific technology classes during 
the school day in which classroom teachers are expected to collaborate with a technology teacher. 

In the initial study, the secondary teachers, on the other hand, had high Personal concerns, high Collaboration 
concerns, and high Refocusing concerns. They were concerned that technology have a positive effect and not 
a negative effect. They were also wondering whether technology really is a solution; are there other means of 
teaching that would be better than technology? The group interviewed in the spring exhibited these same 
concerns. The secondary teachers demonstrated elevated concerns across the board. 

The process of change can be more successful if the “concerns” of the individual are considered.^ The concept 
of concerns, in this instance, includes the participant’s feelings, preoccupation, thought, and consideration 
toward the integration of technology within the curriculum. The participants in this project have strong 
Personal concerns and strong Impact concerns. They want to know how this will affect how and what they 
teach. They also want to know the potential effect on student learning. In their view, there cannot be any 
continuous effect unless there is continued support for this project beyond this initial implementation. 
Without funding for additional hardware and software, the training the participants will be receiving in these 



* Hall, G. E., & Hord, S. M. (1987) . Change in schools: Facilitating the process. New York; State University of New York Press. 

* Hall, G. E., George, A., & Rutherford, W. (1 979) . Measuring stages of concern about the innovation: A manual for the use of the SoC 
questionnaire (Report No. 3032). Austin: The University of Texas at Austin, Research and Development Center for Teacher 
Education. (ERIC Document Reproduction Service No. ED 147 342) 
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workshops will be of limited usefulness. Additionally, they would like for this opportunity to be afforded to 
other faculty members, and, in their view, without continued financial support this will not be possible. 

Primary among the participants’ concerns is the availability of technology, additional funding, time to 
implement integration projects, time to learn the necessary skills, and technical support. One participant 
expressed the concern that “the technology that we currently have is not being used to its fullest capacity.” 

The reason given for this view focuses primarily on training of faculty and staff. Once they have training, the 
participants are concerned that they will not have the necessary hardware and software to adequately 
integrate technology within the curriculum. Characteristic of this uncertainty was the following statement: 

“At this point in time, I feel quite unprepared and untrained to do much integrating for my students. In 
addition, I don’t believe we have adequate equipment to be able to do a first-rate job of integrating 
technology. So, my question is, will we be able to obtain the equipment needed if we decide to integrate?” In 
relation to time and technical support, the following quote characterizes the participants’ concerns: “Time. I 
am concerned about trying something new and exciting then experiencing glitches beyond my control which 
will waste valuable learning time.”® 

How are teachers currently using technology within the classroom? 

Currently, there is very limited integration of technology within the curriculum. Most integration is limited to 
using technology as a resource tool. Among the administrators, technology is primarily used as an 
administrative tool for preparing memos and administrative reports. One of the administrators, however, 
actively searches for Internet sites and other resources for the teachers in his building. 

The integration of technology within the curriculum by teachers seems to be limited, to a great extent, by the 
availability of up-to-date technology within their classrooms. Those with only one computer in their room use 
the computer primarily as a resource tool for preparing for their classes. Several teachers go somewhat 
further and find Internet sites to enhance their lessons — either at home or on the classroom computer, 
depending on their access to the Internet. 

Those with access to more than one computer utilized a variety of models: word processing by students, drill 
and practice, and the generation of products by students. Those using the computer for word processing by 
students were primarily using the computer as a tool for students to type their final products: reports, poems, 
and stories. Several teachers used the computer as a workbook that would give immediate feedback, 
providing students with further practice by using drill and practice computer programs. Finally, a couple of 
teachers designed specific assignments in which the students used computer software and the Internet to do 
research and eventually develop a product on the computer. Teachers who go to the extent of having students 
create products on the computer hold the view that by using the computer as a generative learning tool 
students will develop their higher-order thinking skills. These teachers also had easy access to three or more 
computers. 
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At what Level of Use are participants in their current use of technology and its integration 
within the curriculum? 

The Levels of Use construct provides a key ingredient for understanding and describing the implementation 
of an innovation. Levels of Use focuses on the behaviors that are or are not taking place in relation to the 
innovation.® Typically a person will move in sequence from Level of Use 0, Non-Use, to Level of Use IVA, 
Routine, assuming that the innovation is appropriate, the leader and other change facilitators fulfill their 
roles, and time is provided. The first use of an innovation tends to be disjointed and erratic. Most new users 
cling to the users guide and concentrate on the day-to-day uses more than considering long-term uses. Hall 
and Hord’° indicate that individuals typically remain at the Mechanical level for an extended period of time. 

As the user becomes quite experienced, they move in to Level of Use IVA, Routine. Once users reach a 
Routine Level of Use they typically fall in to a comfortable pattern for using the innovation. As users move 
toward the higher Levels of Use — IVB, Refinement: V, Integration; and VI, Renewal — adaptations are 
intended to improve the effectiveness and positive outcomes from using the innovation. The focus is now on 
increasing effects with students. 

The participants in this project ranged in their Levels of Use from that of a Non-User, orientation perspective, 
to that of a teacher who was working to integrate her use with that of other teachers in an effort to increase the 
effect of the integration of technology on her students. 

The administrators were primarily at a Mechanical Level of Use, still working their way through the idea of 
integrating technology within the curriculum and developing their own skills to the point where they feel 
comfortable using the technology that is available to them. One administrator was at LoU IVA, Routine. The 
administrator was comfortable with his current use of technology and was not, at that time, looking to change 
how he was using the computer. His use of technology centered on administrative word-processing and using 
the Internet as a means for accessing the news. Another administrator was at LoU LVB, Refinement. This 
administrator spent a good share of time looking for resources, software, and Internet sites that would help 
teachers in his building increase the effect of technology on student learning. 

The school board members who participated in this study were dichotomous in their use of technology. One 
board member was active in the use of technology as an administrative tool for business purposes. The other 
board member was a Non-User, preparing to learn how to use technology. 

The secondary teachers ranged from a Non-User to teachers who had already reached a Routine Level of Use 
or who were refining their current use so that they could have a greater effect on their students. One of the 
secondary teachers actively uses the three computers available to her to teach the content to her students. She 
utilizes the Internet for student research, runs tutorial programs to help her students further develop their 
skills, and develops projects for the students to complete on the computers. Another teacher uses the nine 
computers available to him as a supporting piece for one of the units he teaches. When students use the 
computers it is primarily for the purpose of drill and practice or to complete spreadsheet exercises outlined in 
the text. Access to the computers is limited to keep students from playing games on the computers. 

The elementary teachers ranged from a Non-User to one who was working with other teachers, 
collaboratively, in order to have a greater effect on her students. One of the elementary teachers used 
AlphaSmart®, to which her students have access, for students to type the final draft of their writing. The 
writing is then uploaded to the computer and printed out. This particular teacher is striving to learn all she 



®Hall, G. E., & Hord, S. M. (1987). Change in schools: Facilitating the process. New York: State University of New York Press. 
Hall, G. E., & Hord, S. M. (1 987) . Change in schools: Facilitating the process. New York: State University of New York Press. 
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can to increase the effect of technology on her students, including working with other teachers to enhance the 
learning process. She is at an Integration Level of Use (V). But other elementary teachers were at a Mechanical 
Level of Use (III). 



What technology related skills are students currently acquiring through their teachers use 
of technology within the curriculum? 

A majority of the participants said that students were learning how to use computer application programs 
because the teacher was integrating technology within the curriculum. When questions regarding the use of 
application programs were framed in the statement “Because of my use of technology in the classroom,” 74% 
(71% spring) of the participants said that students could use a word- processing program and 70% (93% 
spring) said that students could print their own documents. Only 52% (50% spring) of the participants said 
that students could use a database. Fifty-seven percent of the participants said their students could use 
HyperStudio® in the spring, compared with only 43% in the fall. The use of other application programs was 
not perceived to be as strong (see Table 2). 



Table 2. Percentage of Participants’ Perception of Stude nt Technology Skills 



Application Program 


Fall 1 Spring 


Students Can Use 


Word Processing 


74% 


71% 


Print Documents 


70% 


93% 


Databases 


52% 


50% 


HyperStudio 


43% 


57% 


Spreadsheets 


39% 


43% 


PowerPoint® 


17% 


7% 



What is the perceived impact of current technology use by teachers on students? 

Participants were asked to respond to a variety of impacts that have been identified as being a result of 
integrating technology within the curriculum. The initial group of participants in this project did not strongly 
agree or agree with very many of the 76 phrases presented to them. Those interviewed in the spring 
maintained a more positive attitude. This may be the result of a year’s worth of work or a difference in 
attitude promoted by the training provided. These phrases represented impacts on the quality of student 
work, learning behaviors, interactive classroom organization, teaching with technology, student interaction, 
and communication. Table 3 details those phrases in which more than 50% of the participants strongly 
agreed or agreed with the statement. Each area covered by the impact survey is represented in Table 3. 
Participants agreed with other statements included in the survey but not to the level identified in the table. 
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able 3. Responses to PhrasesConcernii ^^ in the Classroojn _ 



Because of my use of technology in the classroom: 



FaU 



Spring 



Strongly Agree/ Agree 



Quality of Student Work 




iitudents take rhbre interest in learning. 


48% 


79% 


Students stay focused on learning for longer periods of time. 


39% 


71% 


Students demonstrate more independent learning. 


70% 


64% 


Students use technology to support higher levels of learning. 


48% 


79% 


Students are learning the content in greater breadth. 


30% 


79% 


Students express interest in topics being studied. 


52% 


71% 




Learning Behaviors 




Student work demonstrates more creativity. 


48% 


86% 


The complexity of students’ work has increased. 


43% 


71% 


Students express original thoughts related to classroom topics. 


35% 


71% 


Students are adding relevant information to the classroom 
discussion. 


39% 


71% 


. Interactive Classroom Organization 




I plan more hands-on and cooperative learning activity. 


52% 


43% 


Technology has allowed me to increase the degree of student 
involvement. 


52% 


64% 


b Teaching with Technology 


^ J.- -JK- V.' 


I have learned to use a new technology teaching tool. 


52% 


64% 


The computer has become a learning tool in my classroom. 


52% 


71% 


I have found that my enthusiasm for teaching has increased. 


43% 


71% 


I am able to make content more relevant to students (real world). 


65% 


64% 


I have greater access to outside resources. 


70% 


71% 


I now allot more class time to hands-on learning activities. 


48% 


64% 


I use the computer as a learning center in my classroom. 


52% 


57% 


Technology has increased the reliability-validity of information 
available for me to use in instruction. 


57% 


57% 


Student Interaction 




Students are encouraged to participate in peer tutoring. 


48% 


79% 


I use my students as a resource for working with the computer. 


65% 


64% 




Communication 




I communicate personally with other teachers at my school more. 


52% 


36% 



Summary 

The participants in this project have limited technology skills, and the integration of technology within the 
curriculum is not happening, except on a very limited basis. The majority of the integration techniques that 
are being used have the possibility of a very limited effect on student learning and on how students are being 
taught. Hence, the participants have a very limited view as to the effect that technology is having on their 
teaching and their students. Only a few teachers are currently striving to make technology an integral part of 
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student learning by having students use the computer as a generative learning tool, creating real-world 
products and participating in real-world simulations. 

The administrators participating in the project have limited technology skills and are only at a Mechanical 
Level of Use with the skills that they do possess. The secondary teachers possess greater skill with the 
technology but their use is limited. They are primarily using the computer as a resource, and that, at a 
Mechanical Level of Use. Elementary teachers have a limited skill with the technology but are striving to use 
the technology for more than a resource. They are limited, however, in their use, beyond students doing word 
processing, by their lack of knowledge and lack of access to technology. 

Of greatest consequence are the concerns expressed by the participants. Many of the participants feel that 
they are being pressured by the administration to change computer operating platforms. This has proven to 
be a major block in the dissemination of the project. This was discussed after the interviews conducted in the 
fall, and it continues to be a block. The Information concerns of the participants have not been addressed. 
Participants need and want to be informed about the greater picture of this project. They want to know where 
the project is going. Is the long-term support available so that this project can continue? Are funds available 
to purchase the hardware and software necessary to effectively and efficiently integrate technology within the 
curriculum? Will technical support be available? Will training continue? These Personal concerns could 
greatly interfere with the participants’ ability to take full advantage of this project. Their questions need to be 
answered. 

Recommendations 

Following are a series of recommendations made based upon our conversations with the project participants. 
Many of the participants had unanswered questions and unanswered concerns. All of these are potential 
roadblocks to the success of this project. 

1. We recommend that an informal meetinghe held with the educators participating in this project. 
At this meeting the following issues need to be discussed: 

• What is the “phase-in” approach for this project? 

• What are the plans for expansion? Who is getting what? When? What are the plans for 
continued funding support by the district? 

• Where is this project going? 

• What is the time line for training? What will be taught? 

• How will the participants know when they are finished with training? 

• Integration training? When, where, how? 

2. Share the vision with the participants. Allow time for teachers who are currently integrating 
technology within the curriculum to share some of the activities they do with their students. 
Either provide time in a meeting for a group “show and tell" or, better yet, provide teachers 
release time to actually visit classrooms that are currently integrating technology. 

3. Address the platform issue. There seems to be quite a bit of strife over the mandated shift from 
Macintosh® computers to Windows-based computers. A compromise needs to be reached. 
Perhaps the model that has been implemented in Glendo could be followed. All Macintosh 
computers have been moved to the middle school and the new PCs are going in to the high 
school. It is very possible to run Macs and PCs on the same server — Novel and NT both allow 
this and it is not too difficult. 
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4. Address “access " concerns — all the training in the world is of no use if there are not computers for 
the teachers to use, with students, once they have gained the skills. 

5. Provide integration training on how to work within the realm of the one-computer classroom. 

6. Gain further community support by informing parents about the project that is being 
implemented. 
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